Background

Currently, the replication process is effectively linear / "single threaded". When a new sync rules version is deployed, we create a new replication stream, which performs a snapshot on each table, then starts streaming. This has a couple of limitations:

1. Replication is slower than it needs to be due to not being able to replicate tables concurrently.
2. After the initial table snapshots are complete, there could be a significant replication lag that we need to catch up on.

The changes here are also part of the bigger project to implement differential sync rule updates - only re-replicating for changed bucket definitions / sync stream definitions. Part of that requires switching to a single replication stream for all copies of sync rule versions, and this builds the base to implement that.

Changes to storage implementation

Firstly, this changes the underlying BucketStorageBatch implementations to be safe under concurrent usage. Right now it is still designed to only have one process doing streaming replication with commits at a time, but it is safe to have multiple other processes doing snapshots concurrently.

To implement this, we reduce the reliance on local state, instead using the database state which is safe for concurrent access. While this does not introduce any new model fields yet, we do now rely more strongly on snapshot_done (keeps track of whether or not we're waiting for any snapshots to complete) and keepalive_op (keeps track of ops persisted but not committed yet). This has the specific implication that new slots always need an explicit markAllSnapshotDone() call - this is not done automatically anymore.

This also removes the implementation difference between keepalive(lsn) and commit(lsn) - these now both do the same thing.

Changes to replication

Starting with Postgres, we now start streaming changes immediately when starting replication, even if a snapshot is required. To avoid consistency issues, we:

1. Ignore any rows picked up by streaming replication when we do a snapshot.
2. Implement soft-deletes to similarly prioritize deletes in the replication stream over rows in the table snapshot [PENDING].

This also splits out the snapshot implementation from the streaming replication implementation. The snapshotter keeps a queue of tables to snapshot. Currently it only snapshots one at a time, but we can change this in the future.

Tasks

• Apply the same changes to Postgres bucket storage.
• Re-implement the createEmptyCheckpoints logic.
• Get all current tests passing.
• Implement soft deletes, and add tests for that edge case.
• Add low-level tests for concurrent storage usage.
• Proper "job management" for the various replication tasks.
• Docs.
• MongoDB and MySQL implementations (future PR).